<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html
     PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>LingPipe Tracker Tutorial</title>
<meta http-equiv="Content-type"
      content="application/xhtml+xml; charset=utf-8"/>
<meta http-equiv="Content-Language"
      content="en"/>
<style type="text/css" media="screen,projection">
* {
  margin: 0;
  padding: 0;
  font-family: verdana, arial, sans-serif;
}
img { 
  margin: 0.75em 2em 0.75em 2em;
 }
b { 
  font-weight: bold;
}
i { 
  font-style: italic;
}
.yellowbg { 
  background-color: #FFFFEE;

}
.bluebg { 
  background-color: #EEF;
}
body { 
  font-size: 1em;
  background-color: #272651;
}
.var { 
  font-style: italic;
}

a { 
  color: #5756B1; 
  text-decoration: underline;
}
a:hover { 
  color: #272651;
}

pre {
  font-family: "lucida console", "courier new", courier,  monospace;
  font-size: 1.1em;
}
pre.code { 
  padding: 0.5em 1.5ex;
  margin: 0.85em 2ex .85em 1.5ex;
  color: black;
  background-color: #F8F8F8;
  border-top: 1px solid #DDD;
  border-bottom: 1px solid #DDD;
  overflow: auto;
  clip: auto;
  line-height: 130%;
  font-size: .75em;
}
pre.license { 
  padding: 10em;
  font-size: 190%;
}
.greycode { 
  font-family: "lucida console", "courier new", courier,  monospace;
  color: #777;
}

.userinput { 
  font-family:  "lucida console", "courier new", courier,  monospace;
  font-style: italic;
}

div.sidebar pre.code { 
  font-size: .80em;
}
div.author { 
  padding: 0.5em 0ex 0.25em 1.2ex;
}

table { 
  background-color: #CCC;
  margin: 0.75em 2em 0.75em 2em;
  overflow: auto;
  margin-right: 20%;
}
th.title { 
  margin: 0;
  padding: 0.5em 0 0.5em 1ex;
  text-align: left;
  font-weight: bold;
  font-style: normal;
  background-color: #DDD;
  color: black;
  border: none;
}
th, td { 
  margin: 0;
  padding: 0.25em .9ex;
  text-align: left;
  line-height: 130%;
  font-style: normal;
  font-weight: normal;
}
th { 
  font-size: .80em;
  background-color: #E8E8E8;
}
td { 
  font-size: .75em;
  background-color: #FFF;
}
code, .code { 
  font-family: "lucida console", "courier new", courier,  monospace;
  font-size: 1.00em;
}
div#content code { 
  font-family: "lucida console", "courier new", courier,  monospace;
  font-size: 1.15em;
}
.smallcode { 
  font-family: "lucida console", "courier new", courier,  monospace;
  font-size: .90em;
}
.smallnote { 
  color: #666;
  font-size: .90em;
  font-weight: normal;
}

div#header {
  color: white;
  margin: 0 1% 0 0;
  padding: 5px 0 0 0;
}
h1#product {
  float: left;
  width: 19%;
  margin-left: 1%;
  margin-top: .6em;
  margin-bottom: 1px;
  font-weight: normal;
  font-size: 1.50em;
  display: inline;
  color: #F7EB00;
  overflow: hidden;
}
h1#pagetitle {
  float: left;
  font-size: 2.50em;
  margin: 0;
  font-weight: normal;
}
a#logo { 
  float: right;
  padding: 0;
  margin: 0;
  display: inline;
}
a#logo img { 
  border: none;
  padding: 0;
  margin: 0;
}
div#navig {
  clear: both;
  float: left;
  width: 18%;
  margin-left: 1%;
  margin-right: 1%;
  overflow: hidden;
  display: inline;
}
div#navig ul {  
  list-style-type: none; 
  border-bottom: 1px solid #F7EB00;
}
div#navig ul li { 
  border-top: 1px solid #F7EB00;
  color: white;
  font-size: .85em;
  padding-top: 0.6em;
  padding-bottom: 0.6em;
}
div#navig ul li ul {  
  list-style-type: square;
  border: none;
  padding: .15em 0 0 2.5ex;
}
div#navig ul li a { 
  color: white;
  text-decoration: none;
}
div#navig ul li a:hover { 
  color: #F7EB00;
}
div#navig ul li ul li {
  border: none;
  padding: .1em 0 .2em 0;
}
div#navig ul li ul li a {
  font-size: 1.10em;
}
div#navig ul li ul li ul { 
   list-style-type: circle;
}
div#navig a.current {
  color: #F7EB00;
}
div#navig a.current:hover {
  text-decoration: none;
  color: #AAA;
}


button.plainbutton { 
   padding: 5px;
   margin: 3px 3px 3px 0;
}

div.search { 
  text-align: left;
  margin-top: 1.5em;
  width: 100%;
}
input.query { 
  margin: 0 0 2px 0;
  padding: 1px;
  font-size: .80em;
  width: 90%;
  background-color: #DDD;
}
input.submit { 
  font-size: .70em;
  font-family: verdana;
  border: 1px solid #AAA;
  background-color: #272651;
  color: white;
  padding: 1px;
}
input.submit:active { 
  background-color: #888;
  color: white;
  border: 1px solid black;
}
input.submit:hover { 
  color: white;
  border: 1px solid #F7EB00;
}

div#content {
  width: 77%;
  float: left;
  padding: 0 .5em 1.75em .5em;
  font-size: .90em;
  color: black;
  background-color: white;
  overflow: hidden;
}
div#content h2 {
  clear: both;
  font-size: 1.20em;
  margin: 0.5em 0 0.25em 0;
  padding: 0.1em 0 0.25em 1.05ex;
  font-weight: bold;
  background-color:#272651;
  color: white;
}
div#content h3 {
  margin: 0.7em 0 0em 1.4ex;
  font-size: .90em;
  font-weight: bold;
  color: black;
}
div#content h4 {
  margin: 0.75em 0 0em 1.4ex;
  font-size: .9em;
  font-style: italic;
  font-weight: normal;
  color: black;
}
.inhead { 
  font-weight: normal;
  font-style: italic;
  padding: 0 0 0 .1ex;
  margin: 0;
}
.inhead:after { 
  content: ":";
}
div#content blockquote { 
  margin: .6em 3ex .6em 5ex;
}
div#content p {
  margin: 0.5em 20% .5em 1.4ex;
  font-size: .80em;
  line-height: 150%;
}
div#content table tr td ul { 
  margin: .25em 4ex;
  font-size: 1.00em;
}
div#content ul {
  list-style-type: square;
  padding: 0 0 0 0;
  margin: 0em 25% .5em 4ex;
  color: black;
  border-bottom: none;
  border-top: none;
  line-height: 140%;
}
div#content ul li {
  margin: 0.25em 0 .25em 0;
  font-size: .85em;
  line-height: 140%;
}
div#content ul li code {
  font-family: "lucida console", "courier new", courier,  monospace;
  font-size: 1.0em;
}


div#content div.sidebar { 
  float: right;
  clear: right;
  display: inline;
  width: 40%;
  margin: 0.25em 0 0.75em 1em;
  padding: .25em 0.5ex .5em 1.5ex;
  border-top: 1px solid #272651;
  border-bottom: 1px solid #272651;
  font-size: 1.00em;
  color: white;
  background-color: #EEE;
}
div#content div.sidebar h2 { 
  padding: 0;
  margin: .1em 0 .25em 0;
  font-size: .90em;
  font-weight: bold;
  color:#272651;
  background-color: #EEE;
}
div#content div.sidebar p { 
  margin: 0.5em 2ex 0 0;
  padding: 0 1ex 0 0;
  font-size: .75em;
  background-color: #EEE;
  color: black;
}
div#content div.sidebar ul {
  list-style-type: square;
  color: black;
  border-bottom: none;
  border-top: none;
  margin: 0.5em 2ex 0.5em 3ex;
  font-size: .85em;
}
div#content div.sidebar ul li a { 
  color: #5756B1; 
  text-decoration: underline;
  font-size: 1.00em;
}
div#content div.sidebar ul li a:hover { 
  color: #272651;
}


dt {
  display: block;
  margin-top: .5em;
  padding: 0 0;
  color: #333;
  font-weight: bold;
  font-size: .75em;
}


dd {
  color: black;
  font-size: .75em;
  width: 100%;
}

div#foot {
  clear: both;
  width: 97%;
  display: inline;
}
div#foot p { 
  float: right;
  padding: .1em 1.5% 1.5em 0;
  text-align: right;
  font-size: .60em;
  color: #999;
}
div#foot a { 
  padding: 0;
  margin: 0;
  font-size: 1.10em;
  color: #999;
}

div#contentnav { 
  width: 97%;
  margin: 0;
  padding: 0;
}

div#breadcrumbs { 
  float: left;
  margin: 5px 0;
  background-color: #272651;
  color: white;
  font-size: .80em;
}

div#breadcrumbs a { 
  color: white;
  background-color: #272651;
}

div#breadcrumbs a.current {
  color: #F7EB00;
}
div#breadcrumbs a.current:hover {
  text-decoration: none;
  color: #BBB;
}

div#contentmenu { 
  margin: 5px 0 0 0;
  float: right;
  color: white;
  background-color: #272651;
  font-size: .90em;
  border-right: 1px solid #F7EB00;
}

div#contentmenu a { 
  padding: 0 1ex;
  color: white;
  border-left: 1px solid #F7EB00;
  margin: 0;
}

div.column1of2 { 
  margin-top: 0;
  float: left;
  width: 49%;
  padding-top: 0;
  margin-bottom: 1em;
  padding-right: 0;
  margin-right: 0;
}

div.column2of2 { 
  margin-top: 1em;
  width: 49%;
  margin-left: 49.5%;
  border-left: 1px solid #AAA;
  padding-top: 0;
  padding-left: 3%;
  margin-bottom: 1em;
}
</style>
</head>

<body>
<div id="header">
<h1 id="product">LingPipe</h1>
<h1 id="pagetitle">Tracker Tutorial</h1>
<a id="logo" href="http://www.alias-i.com/"><img src="http://www.alias-i.com/lingpipe/web/img/logo-small.gif" alt="alias-i logo"></a>
</div><!-- head -->

<div id="navig">
<ul>
<li><a href="http://www.alias-i.com/lingpipe"><b>LingPipe Home</b></a></li>
<li><a href="http://www.alias-i.com/lingpipe/web/sandbox.html"><b>Sandbox</b></a></li>
</ul>
<br />
<ul>
<li style="font-size:small">This project resides in the LingPipe Sandbox.  
Visit the LingPipe home page or sandbox page, linked above, for more information.
</li>
</ul>
&nbsp;
</div>
<div id="content" class="content">



<h2>Introduction</h2>

<p>
This tutorial covers the Tracker sandbox application.  Trackers
showcase several LingPipe technologies in an application that
collects mentions of entities in documents and links them
across a document collection.  The tracker also accepts a list
of known entities and aliases, and links newly found mentions
of entities in text to the list of known entities.
</p>

<h3>What's a Tracker?</h3>
<p>
A tracker provides a mean of extracting textual mentions of
named entities from documents and linking them together
both within and across documents.  It allows users to specify
a collection of known entities by providing aliases for them.
</p>
<p>This page covers how to run the tracker application only. To
understand how we compute coreference you need to look at the <a
href="coreference.html">within document coreference documentation</a>
and the <a href="xdoc.html">cross document coreference
documentation</a>.
<p>
Trackers are accessible either through a Java API, or
through an HTTP-based web service.  The API and web
service both support the two primary tracker functions:
</p>
<ul>
<li>Analyze Documents: Accept a collection of documents in and process them
linguistically, returning a representation of their entity mentions and their
cross-document identifiers and types.
</li>
<li>Update Dictionary: Send an updated dictionary of known entities,
including aliases.
</li>
</ul>

<h3>Tracker Web Service</h3>
<p>
The tracker web service runs trackers through servlets.  This
means that they are accessed via HTTP, just like web pages.
</p>
<p>
The web services all use XML for input and output. There is
no restriction on character sets for input.  All output is
provided in the UTF-8 encoding of the Unicode character set.
</p>
<p>
The web service saves all documents it analyzes and time
stamps them.  It also saves the last dictionary it received.
This allows the tracker web service to be initialized from
disk.  This supports the servlet lifecycle by allowing the
servlet container to destroy a servlet after inactivity and
reinitialize it as needed.  It also supports the following
additional operation in the web service:
</p>
<ul>
<li>Expire Documents: Send a time such that all documents before the
specified time are expired.
</li>
</ul>


<h3>Trackers and LingPipe</h3>

<p>
The trackers defined in this project are based on several LingPipe packages:
</p>
<ul>
<li><code>com.aliasi.coref</code>: Within-document entity mention coreference</li>
<li><code>com.aliasi.chunk</code>: Chunking (sentences, mentions, entities)</li>
<li><code>com.aliasi.dict</code>: Dictionaries (known entity mention chunking)</li>
<li><code>com.aliasi.sentences</code>: Sentence Detection</li>
<li><code>com.aliasi.hmm</code>: Hidden Markov Models (speculative entity mention detection)</li>
<li><code>com.aliasi.tokenizer</code>: Tokenization</li>
<li><code>com.aliasi.xml</code>: XML Parsing and Generation</li>
</ul>


<h2>XML Document Formats</h2>

<p> Extensible markup language (XML) documents are used for
communication in the web service, and may also be used through the
API.  We first describe these formats and their intended use.  We then
turn to how to access the web service and API.  </p>

<h3>Well-Formedness and DTDs</h3>

<p>
Each document format is supplied with a document type definition (DTD).
These DTDs should be referenced from the document.  All documents
must be well-formed or the API will raise exceptions or the
web service will return error codes.
</p>

<h3>Character Encodings</h3>

<p>
Each XML document should specify its character encoding.  In the API,
this encoding should be set on the input source. In the web service,
the character encoding should be specified in the HTTP request headers.
</p>
<p>
If character encodings are not specified on documents, the
XML parser (Xerces, by default in Sun's Java distributions) will
do its best to automatically detect the character set through
its usual means.
</p>


<h3>Case Sensitivity</h3>

<p>
All aliases, mentions, stoplist entries, and all other text content
is case sensitive in all of the tracker documents.  This allows
&quot;Bush&quot; the person name to be separated from the plant of the same
name.  But it also means that if there is naturally occurring case variation,
it must be included explicitly in lists.
</p>
<p>
XML itself is case sensitive, so all elements and attributes must be
used in their defined case.  
</p>


<h2>User-Defined Entity Dictionary</h2>
<p>
The user may submit a known entity dictionary.  Such a dictionary
is organized by entity, and each entity is provided with a unique
long integer identifier, a type, and a set of aliases.  
</p>

<h3>Example Entity Dictionary</h3>
<p>
Here's an example of a user-defined entity dictionary:
</p>
<pre class="code">
&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;!DOCTYPE dictionary
          PUBLIC &quot;alias-i-entity-dictionary&quot;
          &quot;http://www.alias-i.com/dtd/entity-dictionary.dtd&quot;&gt;

&lt;dictionary&gt;
  
&lt;entity id=&quot;1012&quot; type=&quot;MALE&quot; canonical=&quot;George Bush&quot; speculativeAliases=&quot;0&quot;&gt;
  &lt;alias&gt;George W. Bush&lt;/alias&gt;
  &lt;alias&gt;George Walker Bush&lt;/alias&gt;
  &lt;alias&gt;George W Bush&lt;/alias&gt;
  &lt;alias xdc=&quot;0&quot;&gt;George Bush&lt;/alias&gt;
  &lt;alias xdc=&quot;0&quot;&gt;Bush&lt;/alias&gt;
  &lt;non-alias&gt;George H. W. Bush&lt;/non-alias&gt;
  &lt;non-alias&gt;George Herbert Walker Bush&lt;/non-alias&gt;
  &lt;non-alias&gt;George H W Bush&lt;/non-alias&gt;
  &lt;non-alias&gt;George H.W. Bush&lt;/non-alias&gt;
  &lt;non-alias&gt;George HW Bush&lt;/non-alias&gt;
&lt;/entity&gt;

&lt;entity id=&quot;1013&quot; type=&quot;MALE&quot; canonical=&quot;George H. W. Bush&quot;&gt;
  &lt;alias&gt;George H. W. Bush&lt;/alias&gt;
  &lt;alias&gt;George Herbert Walker Bush&lt;/alias&gt;
  &lt;alias&gt;George H W Bush&lt;/alias&gt;
  &lt;alias&gt;George H.W. Bush&lt;/alias&gt;
  &lt;alias&gt;George HW Bush&lt;/alias&gt;
  &lt;alias xdc=&quot;0&quot;&gt;George Bush&lt;/alias&gt;
  &lt;alias xdc=&quot;0&quot;&gt;Bush&lt;/alias&gt;
  &lt;non-alias&gt;George W. Bush&lt;/non-alias&gt;
  &lt;non-alias&gt;George W Bush&lt;/non-alias&gt;
  &lt;non-alias&gt;George Walker Bush&lt;/non-alias&gt;
&lt;/entity&gt; 

&lt;stoplist&gt;
  &lt;phrase&gt;A&lt;/phrase&gt;
  &lt;phrase&gt;Finally&lt;/phrase&gt;
&lt;/stoplist&gt;

&lt;/dictionary&gt;
</pre>

<h4>XML Declaration</h4>

<p> The document begins with the XML declaration.  Here we have
included a character encoding declaration (<code>UTF-8</code>), which
should match the character encoding used to encode the document.  </p>

<h4>DTD Declaration</h4>

<div class="sidebar">
<h2>Public DTDs</h2>
<p>
To make it easy to browse documents, we've provided
a public URL for the DTD.  Alias-i does not guarantee
up time for this DTD.  We would greatly appreciate it
if you don't contually hit our server for the DTD in
an application.  To that end, the tracker builds it
into the document handler for the SAX parser.
</p>
</div>

<p> The second component of the dictionary is the document type
definition (DTD).  Recall that the first item is the top-level element to be
found in the document (<code>dictionary</code>).  After the public
(<code>PUBLIC</code>) tag is the name of the DTD
(<code>alias-i-entity-dictionary</code>).  This name
may be used internally in programs to provide the DTD to a document
parser (see the tracker code for an example of how to do this in
Java for SAX).  There is also a public URL (<code>http://...</code>).  This
is where the DTD may be downloaded from the web (see the
sidebar for more information).
</p>

<h4>Top-Level Element</h4>

<p>The XML content itself consists of a single element, matching
the specification in the document type declaration (<code>&lt;dictionary&gt;</code>)
The dictionary element is simply a top-level container for entity definitions
and a list of stop phrases.
</p>

<h4>Entity Specifications</h4>

<p>Entities are specified with the <code>entity</code> element.  The
example above has two entity specifications, one for George W. Bush,
and one for his father, George H. W. Bush.  The entity specification
element has several attributes.  The attribute (<code>id</code>)
specifies a unique long-integer identifier for the entity.  For
instance, George W. Bush has identifier <code>1012</code>.
This
identifier will be used by the application to uniquely identify
mentions of the specified entity in documents.  
</p>
<p>
The identifiers below one billion (1,000,000,000)
are reserved for use for user-defined entities.  Identifiers above that
number should only be included in dictionaries if the entity was
returned by the system with that identifier.  For example, after
running the system and submitting documents, the entity &quot;New York&quot;
is found as a location and assigned identifier 1000000174.  At this
point, if an entity for New York is added to the dictionary, the same
identifier should be used.
</p>

<p>The second attribute
(<code>type</code>) provides a type for an entity.  In both entities
in this example, the type is <code>MALE</code>.  The types that are
found by speculative entity detection (see below) are:
</p>

<ul>
<li><code>PERSON</code></li> 
<li><code>MALE</code></li>   
<li><code>FEMALE</code></li> 
<li><code>LOCATION</code></li>
<li><code>ORGANIZATION</code></li> 
</ul>

<p>
These types should be used for entities of the specified type.
Other types may be used, and they will be found by the dictionary-based
entity detector (see below).
</p>

<p>
Next, there is a canonical name attribute for an entity.  This is the
name that will be used for display purposes for the entity.  Finally,
there is a flag (<code>speculativeAliases</code>), incidcating whether
or not the tracker should try to extend the set of aliases by fuzzy
matching within documents.  This may lead to wider coverage, but may
also introduce errors.  The default value is <code>1</code>, to allow
speculative aliases.  It's generally a good idea to allow speculative
aliases unless they are specifically causing problems for an entity.
</p>
<p>
The content of the entity element is a list of aliases.  Each
alias appears as the text content of the <code>alias</code> element.
For instance, <code>Bush</code> and <code>George W Bush</code> are
both synonyms for the entity 1012.  Each alias comes with an
attribute (<code>xdc</code>) which indicates whether the alias
is enough to uniquely identify the entity.  The default value
is <code>1</code>.  Thus the alias &quot;George H. W. Bush&quot;
is taken to uniquely identify entity 1013, the alias &quot;George Bush&quot;,
which is shared by both entities, is not enough.  Typically, 
if a long form (e.g. &quot;George W. Bush&quot;) is used in a document
and the document later contains a shorter form (e.g. &quot;Bush&quot;),
they will be linked together by the within-document coreference
module (see below).
</p>
<p>
In addition there is a <code>non-alias</code> that explicitly prohibits that alias being linked to the entity, but it does allows for the alias to be a part of another entity. This is distinct from the <code>stoplist</code> which prohibits the phrase from ever being a part of any entity.
</p>

<h4>Stoplist</h4>
<p>
The dictionary also contains a list of phrases that should <em>not</em> be
recognized as entity mentions. This list will be used to restrict speculative
mentions by removing any found by the statistical model that appear on this
list.  This allows systematic speculative mention detection errors to be
corrected by brute force.  These often arise because of document formatting
errors (e.g. capitalized javascript variables or entity escapes),
and boilerplate text (e.g., a news article publisher, which may not be
relevant to the domain).
</p>
<p>
The stoplist itself contains the top-level element <code>stoplist</code>,
which may contain any number of <code>phrase</code> elements, the contents
of which are the stopped phrases.
</p>


<h3>DTD for Entity Dictionaries</h3>
<p>
The DTD for the entity dictionary is as follows:
</p>

<pre class="code">
&lt;!ELEMENT dictionary (entity*,stoplist?)&gt;
  
&lt;!ELEMENT entity (alias+,non-alias*)&gt;
&lt;!ATTLIST entity
          id ID #REQUIRED
          type CDATA #REQUIRED   
          canonical CDATA #REQUIRED
          speculativeAliases CDATA &quot;1&quot;&gt;
  
&lt;!ELEMENT alias (#PCDATA)&gt;
&lt;!ATTLIST alias
          xdc CDATA &quot;1&quot;&gt;

&lt;!ELEMENT non-alias (#PCDATA)&gt;

&lt;!ELEMENT stoplist (phrase*)&gt;

&lt;!ELEMENT phrase (#PCDATA)&gt;
</pre>


<h2>Input Documents</h2>

<p>
The web service or API process documents.  Either accepts documents
in XML format, and the API also accepts object representations of
documents.  The basic interface for the web service is a collection of 
multiple documents.  
</p>

<h3>Example Input Documents</h3>

<p>
Here's a short example input containing two short documents:
</p>

<pre class="code">
&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;!DOCTYPE docs
          PUBLIC &quot;alias-i-tracker-input-documents&quot;
          &quot;http://www.alias-i.com/dtd/input-documents.dtd&quot;&gt;

&lt;docs&gt;

&lt;doc id=&quot;doc653b&quot;&gt;
&lt;title&gt;Nokia Launches New Handsets&lt;/title&gt;
&lt;content&gt;
By CASSELL BRYAN-LOW Nokia Corp. unveiled several new music-focused handsets and a music-downloading service, as the world's largest cellphone maker arms itself against Apple Inc.'s much ballyhooed iPhone.
&lt;/content&gt;
&lt;/doc&gt;

&lt;doc id=&quot;doc653c&quot;&gt;
&lt;title&gt;Google CFO Out&lt;/title&gt;
&lt;content&gt;
BURLINGAME, CALIF. -

How do you play a game when there's no way to win? You don't play at all. Google announced Tuesday that Chief Financial Officer George Reyes will retire as it launches a search for his successor. 
&lt;/content&gt;
&lt;/doc&gt;

&lt;/docs&gt;
</pre>

<div class="sidebar">
<h2>Naming Overload</h2>
<p>
We should be taken to task for using an XML element with tag <code>entity</code>.
That's because the term &quot;entity&quot; has a technical meaning in XML,
namely as the bit between the ampersand (<code>&amp;</code>) and
semicolon (<code>;</code>) in escapes such as <code>&amp;quot;</code>.
</p>
<p>
The second egregious naming overload is with the element tag <code>content</code>.
The term &quot;content&quot; has a technical meaning in XML, too.  It's
the term applied to all of the stuff that comes between the start element
and end element.  
</p>
</div>


<p>
Like the entity dictionary, this example begins with an XML declaration,
including character encoding.  It also follows the entity dictionary example by
providing a DTD declaration, indicating the top level element (<code>docs</code>),
the DTD name (<code>alias-i-tracker-input-documents</code>) and a public
URL.
</p>

<p>
The XML content consists of a top-level wrapper element (<code>docs</code>),
containing any number of document elements (<code>doc</code>).
</p>

<p>
Each document indicates three pieces of information.  First, a unique
document identifier, which may be an arbitrary string.   The ID is specified
as the value of the document identifier attribute (<code>id</code>).  The ID
is required.  Each document has an optional title; if no title is specified,
it is treated as the empty string throughout the application.  The title
is specified as the text content of the title element (<code>title</code>).
Finally, a document has optional additional text, specified as 
the text content of the content element (<code>content</code>).
</p>

<h3>DTD for Input Documents</h3>

<p>
The DTD for the input documents is as follows.
</p>

<pre class="code">
&lt;!ELEMENT docs (doc*)&gt;

&lt;!ELEMENT doc (title?,content?)&gt;
&lt;!ATTLIST doc
          id CDATA #REQUIRED&gt;

&lt;!ELEMENT title (#PCDATA)&gt;

&lt;!ELEMENT content (#PCDATA)&gt;
</pre>


<h2>Output Documents</h2>
<p>
As a result of processing input documents, the web service
returns results in an XML format that is simply an enriched
version of the input document format.  Specifically, the
title and content text is broken into sentences, and any
mentions found in the sentences are wrapped in an element.
</p>

<h3>Example Output Documents</h3>

<p>The following XML is an example returned by the web service
for document processing (all speculatively found entities are marked "OTHER" on htis particular config):
</p>

<pre class="code">

&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;

&lt;docs&gt;
&lt;doc id=&quot;doc653b&quot;&gt;
&lt;title&gt;&lt;s index=&quot;0&quot;&gt;&lt;entity id=&quot;1000000008&quot; start=&quot;0&quot; type=&quot;OTHER&quot;&gt;Nokia&lt;/entity&gt; Launches &lt;entity id=&quot;1000000009&quot; start=&quot;15&quot; type=&quot;OTHER&quot;&gt;New Handsets&lt;/entity&gt;&lt;/s&gt;&lt;/title

&gt;&lt;content&gt;&lt;s index=&quot;1&quot;&gt;By &lt;entity id=&quot;1000000010&quot; start=&quot;3&quot; type=&quot;OTHER&quot;&gt;CASSELL BRYAN-LOW Nokia Corp.&lt;/entity&gt; unveiled several new music-focused handsets and a music-downloading service, as the world's largest cellphone maker arms itself against &lt;entity id=&quot;1000000011&quot; start=&quot;169&quot; type=&quot;OTHER&quot;&gt;Apple Inc.&lt;/entity&gt;'s much &lt;entity id=&quot;1000000012&quot; start=&quot;187&quot; type=&quot;OTHER&quot;&gt;ballyhooed iPhone&lt;/entity&gt;.&lt;/s&gt;

&lt;/content&gt;&lt;/doc&gt;&lt;doc id=&quot;doc653c&quot;&gt;&lt;title&gt;&lt;s index=&quot;0&quot;&gt;&lt;entity id=&quot;1000000013&quot; start=&quot;0&quot; type=&quot;OTHER&quot;&gt;Google CFO Out&lt;/entity&gt;&lt;/s&gt;

&lt;/title&gt;&lt;content&gt;&lt;s index=&quot;1&quot;&gt;BURLINGAME, CALIF. &lt;entity id=&quot;1000000014&quot; start=&quot;19&quot; type=&quot;OTHER&quot;&gt;-

How&lt;/entity&gt; do you play a game when there's no way to win?&lt;/s&gt;

&lt;s index=&quot;2&quot;&gt;You don't play at all.&lt;/s&gt;

&lt;s index=&quot;3&quot;&gt;&lt;entity id=&quot;1000000015&quot; start=&quot;0&quot; type=&quot;OTHER&quot;&gt;Google&lt;/entity&gt; announced Tuesday that &lt;entity id=&quot;1000000016&quot; start=&quot;30&quot; type=&quot;OTHER&quot;&gt;Chief Financial&lt;/entity&gt; Officer &lt;entity id=&quot;1000000017&quot; start=&quot;54&quot; type=&quot;OTHER&quot;&gt;George Reyes&lt;/entity&gt; will retire as it launches a search for his successor.&lt;/s&gt;

&lt;/content&gt;&lt;/doc&gt;&lt;/docs&gt;

</pre>

<p> Other than for the differeing DTD declaration, the only difference
between the output doucment example and input document example is the
addition of sentence and entity markup.  
</p>

<h3>Sentences</h3>
<p>
The text in the title and content elements is marked up for sentence boundaries.
Each sentence is enclosed in its own sentence element (<code>s</code>).  Only
text within sentence boundaries is further analyzed.</p>
<p>
the
sentences within a document are indexed with their position using
attribute <code>index</code>.  The indexing begins with zero
(<code>0</code>).  The numbering is restarted for each document,
but the numbering crosses the title and content elements within
a document.
</p>

<h3>Entities</h3>
<p>
Within sentences, some phrases are marked up as entities using the entity
element (<code>entity</code>).  The entity element has two attributes, one
for a unique long-integer identifier (<code>id</code>), and one for the
entity type (<code>type</code>).  These attributes correspond to those described
in the entity dictionary above.  In fact, for entities in the dictionary,
the identifiers returned in output documents will match the input
dictionary.
</p>
<p>
Examples of entities in the extracted document, include &quot;Nokia&quot;,
which is marked in the title of the first document as an organization
with identifier 38445.  Note that this is the same identifier as assigned
to the phrase &quot;Nokia Corp.&quot; in the content of the first document.
Further note that the pronoun &quot;his&quot; in the second document has
been analyzed as being coreferent with the proper name &quot;George Reyes&quot;.
</p>
<p>
Like the sentences, entities are indexed as to position.  The indexes
for entities are scoped within sentences, so each sentence begins
numbering entity mentions from zero (<code>0</code>).

<h3>DTD for Output Documents</h3>

<p>
The DTD for output documents is similar to the one for
input documents, but it allows sentence elements within
title and content, and sentences may contain entities.
</p>

<pre class="code">
&lt;!ELEMENT docs (doc*)&gt;

&lt;!ELEMENT doc (title?,content?)&gt;
&lt;!ATTLIST doc
          id CDATA #REQUIRED&gt;

&lt;!ELEMENT title (#PCDATA|s)*&gt;

&lt;!ELEMENT content (#PCDATA|s)*&gt;

&lt;!ELEMENT s (#PCDATA|entity)*&gt;
&lt;!ATTLIST s
             index #REQUIRED&gt;

&lt;!ELEMENT entity (#PCDATA)&gt;
&lt;!ATTLIST entity
	  index CDATA #REQUIRED
          id CDATA #REQUIRED
          type CDATA #REQUIRED&gt;
</pre>

<h2>Accessing the Web Service</h2>

<p>
The tracker web service is accessed via the hyper-text transport
protocol (HTTP).  
</p>

<h3>HTTP Request Content Type</h3>
<p>
Part of an HTTP request is a sequence of headers.  These headers
may contain a content-type declaration.  The client must set
the content-type on its HTTP request in order for the service
to be able to decode its input.  
</p>

<h4>Binary XML Requests</h4>

<p>
For straight XML-based POST submissions, the
content type <code>text/xml</code> must be specified.  Content types
optionally allow character encodings to be specified.  The full
request header should look as follows, with the appropriate character
encoding replacing the Latin1 specification in the example (<code>ISO-8859-1</code>).
</p>

<pre class="code">
Content-Type: text/xml; charset=ISO-8859-1
</pre>

<h4>HTML Form-Based Requests</h4>

<p>
The web service is also set up to handle single document input via
a web form.  This service is primarily intended for demonstration
purposes.  Web forms, when submitted to servers by browsers, provide the content
type <code>application/x-www-form-urlencoded</code>.  They further
add keys to the fields to indicate which part of the form created the
input.  Finally, they URL-encode the data they submit.
</p>

<p>To accomodate web form submissions, URL-encoded requests specifying the
following three attributes are accepted (either via POST or GET requests)
to encode documents.
</p>

<ul>
<li><code>id</code></li>
<li><code>title</code></li>
<li><code>content</code></li>
</ul>

<p>
The web service is distributed with an HTML form that carries out this
request.  It may be found in <a href="web/index.html"><code>web/index.html</code></a>.
(Note that it will not be live unless accessed through the service, as it uses
a relative path for submission of its form data.)  
</p>

<p>The skeletal form of the HTML used for the example form is as follows:
</p>
<pre class="code">
&lt;form action=&quot;docs&quot; method=&quot;post&quot;&gt;
  &lt;textarea name=&quot;id&quot;&gt;&lt;/textarea&gt;
  &lt;textarea name=&quot;title&quot;&gt;&lt;/textarea&gt;
  &lt;textarea name=&quot;content&quot;&gt;&lt;/textarea&gt;
  &lt;input type=&quot;submit&quot; value=&quot;Submit Text&quot;&gt;&lt;/input&gt;
&lt;/form&gt;
</pre>
<p>
Note that it specifies the path to provide the web service, <code>tracker/track</code>,
indicates that the POST protocol should be used, and then provides labeled input
fields for the three pieces of text data making up the document.
</p>


<h3>Web Service URLs</h3>


<h4>Base Server URL</h4>
<p>
The web service is launched in a servlet container such as Apache Tomcat.  
The container will determine the base URL for the application.  
For instance, if I run Tomcat locally, using the default install port of 8080,
the base URL for applications is:
</p>
<pre class="code">
http://localhost:8080/
</pre>

<h4>Context Path</h4>

<p>A servlet-based web application is launched within a context.
The context is specified to the container manager when the application
is deployed.  The context determines the path within the server URL
through which the application is accessed.  As we have configued
the application in the Ant build file <a href="build.xml"><code>build.xml</code></a>,
the context is <code>/tracker</code>, and thus all request URLs will
extend:
</p>
<pre class="code">
http://localhost:8080/tracker
</pre>
<p>
The path <code>/tracker</code> may be changed by configuring the
deployment of the servlet.
</p>

<h4>Servlet Path</h4>

<p>
Each servlet is installed in its context with a unique name.  This
name is specified in the deployment descriptor, which may be found
in <a href="web/WEB-INF/web.xml"><code>web/wEB-INF/web.xml</code></a>.
In particulra, each servlet is provided with a name, a class,
and a mapping to a URL pattern.  All requests to the application context
with a URL matching the pattern are handled by the specified servlet.
Here's the servlet specification from the deployment descriptor:
</p>
<pre class="code">
&lt;servlet&gt;
  &lt;servlet-name&gt;TrackerServlet&lt;/servlet-name&gt;
  &lt;servlet-class&gt;com.aliasi.tracker.web.TrackerServlet&lt;/servlet-class&gt;
  &lt;init-param&gt;
    &lt;param-name&gt;save-directory&lt;/param-name&gt;
    &lt;param-value&gt;c:\carp\sandbox\tracker\applicationData\simpleDemo&lt;/param-value&gt;
  &lt;/init-param&gt;
&lt;/servlet&gt;
&lt;servlet-mapping&gt;
  &lt;servlet-name&gt;TrackerServlet&lt;/servlet-name&gt;
  &lt;url-pattern&gt;/tracker/*&lt;/url-pattern&gt;   
&lt;/servlet-mapping&gt;
</pre>
<p>
The most important aspect of this config is the <code>save-directory</code>
initialization parameter for the tracker servlet.  This must point to
an existing, Tomcat-writable directory.  This directory must contain
a valid entity dictionary in a top-level file named 
<code>entity-dictionary.xml</code>.  This will be the initial dictionary
and any new dictionaries submitted will overwrite the old dictionary
without saving a backup (backups are the client's responsibility, as all dictionaries
originate from the client, and our server's not (yet anyway) transactional.).
In addition to the entity dictionary file, the save directory must
contain a subdirectory called <code>docs</code>, which is the directory
to which all saved documents (after processing) are written.  Any documents
in the saved docs directory will be added to the tracker during startup.
</p>
<p>
The servlets live in the <code>com.aliasi.tracker.web</code> package.
There are four of them, each configured on a different
path in the context:
</p>

<p>The following table provides a key to the URLs for the various
web services:
</p>

<table>
<tr><th class="title" colspan="3">URLs for Web Services</th></tr>
<tr><th>URL</th><th>Servlet</th><th>Service</th></tr>
<tr><td><code>http://localhost:8080/tracker/docs</code></td>
    <td><code>DocumentsServlet</code></td>
    <td>Submit Input Documents</td></tr>
<tr><td><code>http://localhost:8080/tracker/dict</code></td>
    <td><code>DictionaryServlet</code></td>
    <td>Submit Dictionary</td></tr>
<tr><td><code>http://localhost:8080/tracker/expire?t=<i>TimeInMsSinceEpoch</i></code></td>
    <td><code>ExpirationServlet</code></td>
    <td>Expire Documents</td></tr>
<tr><td><code>http://localhost:8080/tracker/entities</code>
    <td><code>EntityUniverseServlet</code></td>
    <td>Report entities and aliases</td></tr>
<tr><td colspan="3"></td></tr>
<tr><td><code>http://localhost:8080/tracker/index.html</code></td>
    <td>n/a</td>
    <td>Demo Web Form Input</td></tr>
</table>

<p>In practice, the base URL (<code>http://localhost:8080/</code>)
will likely be different.</p>

<p>Note that the expire documents URL contains a parameter
<code>t</code> whose value is set to the numberof milliseconds since
the epoch (1 January 1970) before which documents should be
expired.</p>



<h2>Building and Installing the Web Service</h2>

<p>
The Ant build file <a href="build.xml"><code>build.xml</code></a> contains
all the information necessary to compile the API, the tracker servlet,
and to deploy them both in the Apache Tomcat servlet container.
</p>

<h3>Building</h3>

<p> The Ant target <code>jar</code> builds the java archive (JAR)
files for the API.  The result is two jars, <code>tracker.jar</code>
and <code>models.jar</code>.  The former contains the API and servlet,
the latter the models used for speculative named entity detection.
</p>

<p> The target <code>war</code> builds the web application archive
(WAR) file.  This is a self-contained version of the tracker web
service which may be deployed to a servlet container.  </p>


<h3>Deploying to Tomcat</h3>

<p>
The <code>tracker.war</code> file and the jars listed in the next
section are required to run the application.  The war file may
be installed as usual, and the jars need to go into the server lib
directory.
</p>

<h4>Jars for Tomcat</h4>
<p>
The following three jars need to go into <code>$CATALINA_HOME/lib</code>:
</p>
<ul>
<li><code>mail.jar</code></li>
<li><code>mailapi.jar</code></li>
<li><code>smtp.jar</code></li>
</ul>

<h4>Deploy Ant Task</h4>
<p>
There are also Ant targets for deploying, undeploying and redploying
the application to Tomcat.  The application may also be deployed
by uploading the WAR through the Tomcat manager web application.
</p>

<p> Handling Tomcat deployment through Ant requires that the
<code>catalina-ant.jar</code> be installed in Ant's <code>/lib</code>
directory.  The jar may be found in Tomcat's <code>/lib</code>
directory.  </p>

<p>There are three relevant Ant targets, <code>deploy</code>,
<code>undeploy</code>, and <code>reploy</code>, the latter of
which is just an undeploy followed by a deploy.  All three
of these tasks depend on the following properties to be set:
</p>

<table>
<tr><th class="title" colspan="3">Ant Deployment Properties</th></tr>
<tr><th>Property</th><th>Default Value</th><th>Description</th></tr>
<tr><td><code>context.name</code></td>
    <td><code>tracker</code></td>
    <td>Name of application context, which is the first part of the web
        service URL after the host and port.  It is also the name by
        which the application is known to the Tomcat manager.</td></tr>

<tr><td><code>tomcat.url</code></td>
    <td><code>http://localhost:8080/</code></td>
    <td>The URL at which Tomcat is running.  The default is the local machine.</td></tr>

<tr><td><code>manager.username</code></td>
    <td><code>admin</code></td>
    <td>The Tomcat administrator's user name, as specified in the Tomcat install (or reconfig).</td></tr>

<tr><td><code>manager.password</code></td>
    <td><code>admin</code></td>
    <td>The Tomcat administrator's password, as specified in the Tomcat install (or reconfig).</td></tr>

<tr><td><code>persist.dir</code></td>
    <td>c:/carp/sandbox/tracker/applicationData/demo/</td>
    <td>The directory containing the dictionary and any
        persisted processed documents.</td>
             
</table>

<p>These may be set in a file parallel to the <code>build.xml</code>
file called <code>user.properties</code>.  Here's an example:
</p>
<pre class="code">
context.name=tracker
manager.username=admin
manager.password=admin
tomcat.url=http://localhost:8080/
persist.dir=C:/carp/sandbox/tracker/applicationData/demo
</pre>

<p>
Note that all of the requisite models, DTDs, etc. are included in the WAR
file.  Nothing else needs to be supplied to the application.
</p>

<h3>Configuration Parameters</h3>
<p>
TBD. (for files used for serialization, possibly interchangeable entity/sentence/tokenizers)
</p>

<h2>Sample Web Service Clients</h2>

<p>
There is a simple Java client program tha demonstrates the client-side
use of the web service in an application.  The source for the program is
in:
</p>
<ul>
<li><a href="src/com/aliasi/tracker/web/TrackerClient.java"><code>src/com/aliasi/tracker/web/TrackerClient.java</code></a></li>
</ul>

<p>
It basically takes a URL, a file from which to gather POST data, a character
set for the post data, and optionally, a file to which to write to the output.
If no output file is specified, the output is written to standard output
(<code>System.out</code>).
</p>

<h3>Documents Client</h3>
<p>
A documents file may be processed using the Ant target <code>docs-client</code>
in the Ant build file <a href="build.xml"><code>build.xml</code></a>.
The target is specified to send requests to the currently deployed tracker
web service (based on the properties <code>tomcat.url</code>, <code>context.name</code>,
and <code>app.name</code>).  It is configured to read from the sample demo input
documents file and write to its sibling (which will not exist until the Ant target
has completed):
</p>
<ul>
<li><a href="applicationData/tutorial-demo/input/docs1.xml"><code>applicationData/tutorial-demo/input/docs1.xml</code></a></li>
<li><a href="applicationData/tutorial-demo/output/docs1.xml"><code>applicationData/tutorial-demo/output/docs1.xml</code></a></li>
</ul>
<p>
Note that the Ant target <code>clean</code> will remove this output directory.
</p>
<p>
The target is run as follows:
</p>
<pre class="code">
&gt; ant docs-client
</pre>


<h3>Dictionary Client</h3>
<p>
A dictionary file may be processed using the Ant target <code>dict-client</code>.
The dictionary target is configured the same way as the documents client target
for the deployed tracker service.  As distributed, it sends the following
dictionary from the demo application:
</p>
<ul>
<li><a href="applicationData/tutorial-demo/entity-dictionary.xml"><code>applicationData/tutorial-demo/entity-dictionary.xml</code></a></li>
</ul>


<h3>Expiration Client</h3>
<p>
Any old web browser may be used to test the expiration client
behavior.  It only involves a URL as specified in the last
section, and no additional HTTP POST data.
</p>


</div><!-- content -->

<div id="foot">
<p>
&#169; 2007 &nbsp;
<a href="mailto:lingpipe@alias-i.com">alias-i</a>
</p>
</div>

</body>

</html>


